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(54) PICTURE PROCESSOR AND PICTURE PROCESSING METHOD 

(57) Abstract: 

PROBLEM TO BE SOLVED: To attain arithmetic processing of a video signal' 
in a real time. 

SOLUTION: Infrared rays made incident to a light receiving part 51 are 
photoelectrically converted synchronously with a reset pulse supplied 
from a timing generatorand outputted to an amplifying part 52 
synchronously with a light receiving part transfer pulse supplied from a 
timing generator. The signal inputted to the amplifying part 52 is 
amplified to a level necessary for processing in the post-device 
synchronously with an amplifying part drive pulse supplied from the 
timing generatorand outputted to an arithmetic part 53. The signal 
inputted to the arithmetic part 53 is temporarily stored in a storage 
part 61and the signal to which a prescribed arithmetic operation 
designated according to an arithmetic operation selection signal from an 
arithmetic operation controlling part is performed is outputted as a 
binary signal from a comparing part 62 to an outputting part 54. The 
signal inputted to the outputting part 54 is outputted through a common 
signal line 42 as a pixel signal synchronously with a selection signal 
from a horizontal scanning circuit. 



CLAIMS 



[Claim (s) ] 

[Claim l]An image processing device which has the optical area which has 
arranged two or more elements to matrix formcomprising: 



A light-receiving means which receives and carries out photoelectric 
conversion of the light which enters for every element of said optical 
area. 

A calculating means which calculates a signal by which photoelectric 
conversion was carried out by said light-receiving means for said every 
element under a predetermined rule. 

An output means which outputs the result of an operation of said 
calculating means for said every element. 

A timing adjustment means which adjusts timing to which the result of an 
operation is outputted from said output means for said two or more 
elements of every. 

[Claim 2]The image processing device according to claim lwherein said 
calculating means includes further a memory measure which memorizes two 
or more signals of said different timing by which photoelectric 
conversion was carried out one by one. 

[Claim 3]The image processing device according to claim 2wherein said 
calculating means performs a comparison operation combining two or more 
signals memorized by said memory measure. 

[Claim 4]The image processing device according to claim 3wherein said 
comparison operation includes an operation which calculates the maximum 
or the minimum of said signal. 

[Claim 5]The image processing device according to claim lwherein said 
output means outputs said result of an operation for every sequence of 
each of said elementor line to timing adjusted with said timing 
adjustment means. 

[Claim 6]An image processing method of an image processing device which 
has the optical area which has arranged two or more elements to matrix 
form characterized by comprising the following. 

A light-receiving step which receives and carries out photoelectric 
conversion of the light which enters for every element of said optical 
area. 

An arithmetic step which calculates a signal by which photoelectric 
conversion was carried out by processing of said light-receiving step 
for said every element under a predetermined rule. 

An output step which outputs the result of an operation of processing of 
said arithmetic step for said every element. 

A timing adjustment step which adjusts timing to which the result of an 
operation is outputted by processing of said output step for said two or 
more elements of every. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]Especially this invention relates to the image 
processing device and method which calculate the signal acquired by 
carrying out photoelectric conversion of the light which receives the 
light from the photographic subject to picturizeand which received light 
for every element about an image processing device and a method under a 
predetermined ruleand could be made to carry out data processing of the 
picture signal in real time. 
[0002] 

[Description of the Prior Art]The art of calculating a picture signal is 
spreading. For example by calculating a picture signalthis art is 
usedwhen asking for the stereoscopic picture of a photographic subject. 
[0003] Although the operation of such a picture signal acquires two or 
more signals required for an operation when using an image sensor like 
CCD (ChargeCoupled Device) After picturizing the repetition photographic 
subject and accumulating them in memory storage like a frame memorythe 
accumulated signal was read and it performed. 
[0004] The "non-scanning image sensor" as shown in JP6-25653B is 
considered as the method of realizing shape measuring of real timeand a 
device. 
[0005] 

[Problem (s) to be Solved by the Invent i on] Howeverwhen picturizing a 
repetition photographic subjectsince one image pick-up would take the 
time of 33.3msec or 16. 6msecthe time which this image pick-up takes 
became a maximumand it had a technical problem that the result of an 
operation of picture information could not be obtained any more at high 
speedfor example. 

[0006] Since the result of an operation was not obtained if it did not 
picturize repeatedly in order to calculate picture informationthe 
technical problem that the result of an operation was not obtained in 
real time occurred. 

[0007] In order to treat independently the output of each pixel located 
in a line on the image sensor in the case of the non-scanning image 
sensor currently indicated by JP6-25653BSince the output signal line 
from each pixel could not be communalized and the memory measure for 



each pixel could not be giving in the element furtherthe feature of "the 
non-scanning-type" that each pixel operates independently was lostand 
the technical problem that processing in real time could not be 
performed occurred. 

[0008]This invention is made in view of such a situationand it is made 
to make the picture signal acquired by carrying out photoelectric 
conversion of the light which receives the light from the photographic 
subject to picturizeand which received light for every element calculate 
in real time in accordance with a predetermined rule. 
[0009] 

[Means for Solving the Problem] written this invention is characterized 
by it having been alike and comprising the following at claim 1. 
A light-receiving means which receives and carries out photoelectric 
conversion of the light which enters for every element of optical area. 
A calculating means which calculates a signal by which photoelectric 
conversion was carried out by a light-receiving means for every element 
under a predetermined rule. 

An output means which outputs the result of an operation of a 
calculating means for every element. 

A timing adjustment means which adjusts timing to which the result of an 
operation is outputted from an output means for two or more elements of 
every. 

[0010]A memory measure which memorizes two or more signals of different 
timing by which photoelectric conversion was carried out one by one can 
be further provided in said calculating means. 
[0011] Said calculating means can be made to perform a comparison 
operation combining two or more signals memorized by a memory measure. 
[0012] An operation which calculates the maximum or the minimum of a 
signal can be made to be included in said comparison operation. 
[0013] The result of an operation can be made to output to said output 
means for every sequence of each elementor line to timing adjusted with 
a timing adjustment means. 

[0014]written this invention is characterized by it having been alike 
and comprising the following at claim 6. 

A light-receiving step which receives and carries out photoelectric 
conversion of the light which enters for every element of optical area. 
An arithmetic step which calculates a signal by which photoelectric 
conversion was carried out by processing of a light-receiving step for 
every element under a predetermined rule. 

An output step which outputs the result of an operation of processing of 



an arithmetic step for every element. 

A timing adjustment step which adjusts timing to which the result of an 
operation is outputted by processing of an output step for two or more 
elements of every. 

[0015] In an image processing device according to claim 1 and the image 
processing method according to claim 6photoelectric conversion of the 
light which entered for every element of optical area is carried outa 
signal by which photoelectric conversion was carried out for every 
element calculates under a predetermined ruleand the result of an 
operation is outputted for every element. 
[0016] 

[Embodiment of the Invent ion] Drawing 1 is a block diagram showing the 1 
embodiment of the image processing device 1 which applied this invention. 
The system control part 11 of the image processing device 1 controls 
operation of the pattern light projection part 12the imager 15the video 
signal processing section 16the distance sensors 17and the formed data 
treating part 18. 

[0017]Based on the instructions from the system control part lithe 
pattern light projection part 12 turns the infrared light of a pattern 
required for distance measurement to the photographic subject 2and 
irradiates with it. As this pattern lightslit lightgrid lightetc. are 
used based on the measuring principle of the distance sensors 17. 
[0018]The lens 13 condenses the light from a photographic subjectand 
introduces it into the prism 14. The prism 14 carries out the spectrum 
of the light which entered from the lens 13 to visible light and 
infrared light. That issince there is catoptric light from the 
photographic subject of the infrared light irradiated from the above- 
mentioned pattern light projection part 12 in the light from a 
photographic subject besides visible lightthe spectrum of this is 
carried out to visible light and infrared lightvisible light is emitted 
to the imager 15 and infrared light is emitted to the distance sensors 
17respectively. 

[0019] The imager 15 consists of CCD (Charge Coupled Device) CMOS 
(Complementary MetalOxide Semiconductor) etc. Based on the synchronized 
signal and control signal from the system control part lithe information 
on a color is extracted from the visible light which entered from the 
prism Hand it outputs to the video signal processing section 16 as a 
video signal. 

[0020]After the video signal processing section 16 performs a gain 
adjustment and color adjustment processing to the video signal inputted 



from the imager 15 based on the synchronized signal and control signal 
from the system control part Hit is changed into an analog signal or a 
digital signal if neededand is outputted to the computer 19 as a color 
video image signal. 

[0021] The distance sensors 17 receive the infrared light which entered 
from the prism 14process this infrared light that received light as a 
binary-ized signal based on the synchronized signal and control signal 
from the system control part Hand output it to the formed data treating 
part 18. About the distance sensors 17details are mentioned later. 
[0022]Based on the synchronized signal and control signal from the 
system control part lithe formed data treating part 18It asks for the 
timing to which the intensity of infrared light serves as a peak from 
the binary-ized signal inputted from the distance sensors 17the distance 
to the photographic subject 2 is computed by the principle of 
triangulation from the intensityand the three-dimensional shape of the 
photographic subject 2 is calculated. And the formed data treating part 
18 is outputted to the computer 19 by making the result of an operation 
into a formed data signal. 

[0023] The color video image signal with which the computer 19 was 
supplied from the video signal processing section 16Computer graphics 
processing is performed to the formed data signal supplied from the 
formed data treating part 18It outputs to the monitor 3 which consists 
of CRT (Cathode Ray Tube) LCD (Liquid Crystal Display) etc. and outputs to 
the external storage 4and the data is stored up. 

[0024]Nextthe details of the distance sensors 17 are explained with 
reference to drawing 2 and drawing 3 . The distance sensors 17 are 
divided roughly and have two types. Are a type which the 1st type scans 
sequentially the pixel 41 located in a line with the horizontal 
direction of the optical area 31takes out the output signal from each 
pixel 41and performs processingand the 2nd typelt is a type which scans 
sequentially the pixel 41 located in a line with the perpendicular 
direction of the optical area 31and takes out and processes the output 
signal from each pixel. Drawing 2 expresses the composition of the 
distance sensors 17 of the former typeand drawing 3 expresses the 
composition of the distance sensors 17 of the latter type. 
[0025]Firstwith reference to drawing 2 the horizontal scanning type type 
distance sensors 17 are explained. 

[0026] The optical area 31 comprises two or more pixels 41 provided with 
the calculation functionandas for the nxm individual (= the number of 
horizontal number x perpendicular directions) the pixel 41 is put in 
order by matrix form. Each pixel 41 calculates the signal according to 



the light volume which received light based on the reset pulse and light 
sensing portion transfer pulse which are outputted from the timing 
generator 32The result of an operation is horizontally outputted to the 
output circuit 34 via the common signal lines 42 based on the selection 
signal supplied from the horizontal scanning circuit 33a. About the 
pixel 41details are mentioned later. 

[0027] The timing generator 32 supplies a control pulse to the horizontal 
scanning circuit 33a and the output circuit 34 based on the control 
signal from the system control part Hand generates an amplifier drive 
pulsea reset pulseand a light sensing portion transfer pulseand supplies 
them to each pixel 41 of the optical area 31. 

[0028]Based on the control pulse supplied from the timing generator 
32the horizontal scanning circuit 33a generates a clear pulse and 
storage parts store transfer pulsea comparing element drive pulseand a 
selection signaland supplies them to each pixel 41 of the optical area 
31. 

[0029] Synchronizing with the control pulse from the timing generator 
32it receives via the common signal lines 42and the output circuit 34 
outputs the output signal from each pixel 41 of the optical area 31 to 
the formed data treating part 18. 

[0030]The operation control part 35 supplies the operation (it 
specifies) selection signal which chooses data processing to perform to 
the matrix circuit 72 ( drawing 6 ) of the storage parts store 61 of the 
operation part 53 of each pixel 41 corresponding to the control signal 
from the system control part 11. About the matrix circuit 72 of the 
storage parts store 61 of the operation part 53details are mentioned 
later. 

[0031] In the distance sensors 17 of the vertical-scanning type type 
shown in drawing 3 It replaces with the horizontal scanning circuit 33a 
of the distance sensors 17 of the horizontal scanning type type shown in 
drawing 2 the vertical scanning circuit 33b is formedand the output of 
each pixel 41 which this drove is perpendicularly supplied via the 
common signal lines 42 in the output circuit 34. Other composition is 
the same as that of the case in drawing 2 . 

[0032]Nextthe details of the pixel 41 are explained with reference to 
drawing 4 . The n pixels 41a thru/or 41n connected to the common signal 
lines 42 shown in drawing 2 are shown in drawing 4 . although only the 
composition of the one pixel 41a is shown here — the other pixels 41b - 
- or 41 n is constituted similarly. This is the same also about the case 
of the distance sensors 17 of drawing 3 . 

[0033]The light sensing portion 51 of the pixel 41a consists of photo 



detectorssuch as a photo-diodefor exampleThe infrared light which enters 
from the prism 14 is receivedphotoelectric conversion of the infrared 
light which received light is carried out based on the reset pulse 
supplied from the timing generator 32and the signal is outputted to the 
amplifier 52 based on the light sensing portion transfer pulse supplied 
from the timing generator 32. 

[0034]The amplifier 52 amplifies the signal inputted from the light 
sensing portion 51 even on a level required for processing with a latter 
device synchronizing with the amplifier drive pulse supplied from the 
timing generator 32and outputs it to the operation part 53. 
[0035]The operation part 53 has the storage parts store 61 and the 
comparing element 62performs the predetermined operation specified by 
the operation selection signal from the operation control part 35 in the 
signal inputted from the amplifier 52and outputs it to the outputting 
part 54 as a binary-ized signal. About the storage parts store 61 and 
the comparing element 62details are mentioned later. 
[0036]The outputting part 54 is outputted to the output circuit 34 
synchronizing with the selection signal from the horizontal scanning 
circuit 33a by making into a pixel signal the signal inputted from the 
operation part 53 via the common signal lines 42. 

[0037]Herethe operation of the binary-ized signal of the operation part 
53 is previously explained in explanation of the storage parts store 61 
of the operation part 53and the comparing element 62. 
[0038] Although the signal corresponding to the received light volume is 
amplified by the amplifier 52 and inputted into the operation part 53 by 
the light sensing portion 51sampling signal [ of the infrared light 
intensity ] s (k) shall change with the time k of a samplingas shown in 
drawing 5 . Whenever the time k changes at this timethe sampling signal 
of infrared light intensity is shown as follows. 

[0039] s (k-3)s (k-2)s (k-l)s (k) s (k-l)s (k-2)and s (k-3) — k-1 assumes 
that the front value is shown in time rather than ....however k. 
[0040]At this timethe function g (k) which shows a displacement 
difference as shown in the following formulas (1) as a function for 
detecting the time when the intensity of infrared light serves as a peak 
is considered. 
[0041] 

g(k) ={s(k)+s (k-1)}- {s(k-2) +s (k-3)} .... (1) 

If this function g (k) supports differentiating sampling signal s (k) 
mostlyand sampling signal [ of infrared light intensity ] s (k) assumes 
it to take a positive big value so that it is largeAt the time of g(k) 
>0it will be shown that sampling signal [ of infrared light intensity ] 



s (k) is increasing to change of timeand it will be conversely shown at 
the time of g(k) <0 that the intensity of sampling signal [ of infrared 
light intensity ] s (k) is decreasing to change of time. 
[0042]Thereforethis g (k) becomes the time when the sampling signal of 
infrared light intensity presents a peak in the time k which changed to 
the negative value from a positive value. 

[0043] By thenthe function f (k) shown in the formula (2) of the 
following which added the bias of the predetermined level to the 
function g (k) in consideration of the noise of sampling signal [ of 
infrared light intensity ] s (k) as shown in drawing 5 . It can ask for 
the time which detected the peak of the sampling signal of infrared 
light intensity by the same technique as the above. 
[0044] 

f(k) ={s(k)+s(k-l)}-{s(k-2) +s(k-3)}+BIAS .... (2) 

the timing which the time k of an intersection with the zero level with 
which f (k) changes corresponding to change of infrared light intensity 
s (k)and the value of f in drawing 5 (k) changes from a positive value 
to a negative value as shown in drawing 5 presents a peak in the 
infrared light intensity s (k) — ****** — it becomes things. Although 
the peak of infrared light intensity s (k) is sampling time (k-2) and 
has a gap to the time k in drawing 5 Since this serves as a fixed gap 
which becomes settled uniquely by the function f (k) it can ask for the 
time k which takes the peak value of sampling signal [ of exact infrared 
light intensity ] s (k) by imposing fixed offset to the computed time. 
[0045] The operation part 53 outputs the binary-ized signal used as 1 to 
the output circuit 34when the value of above-mentioned function f (k) is 
positive or zero and it is 0 and negative. The output circuit 34 is 
outputted to the latter formed data treating part 18 by making this 
binary-ized signal into an output signal. The latter formed data 
treating part 18 searches for the peak of infrared light intensity from 
the time of the sampling used as the peak of the infrared light 
intensity called for by this binary-ized signaland computes the distance 
from this infrared light intensity to a photographic subject by the same 
principle as triangulation. 

[0046]Nextwith reference to drawing 6 the storage parts store 61 and the 
comparing element 62 of this operation part 53 are explained. 
[0047] The storage cells 71a thru/or 71d of the storage parts store 61 
memorize the signal inputted from the amplifier 52 one by one as a 
sampling signal of infrared light intensity based on the clear pulses 
CLR1 thru/or CLR4 sent from the horizontal scanning circuit 33a 
synchronizing with a sampling synchronized signal. 



[0048]Namelyfor examplein a certain time kto the storage cell 71a. To s 
(k) and the storage cell 71bto s (k~l) and the storage cell 71c. 
Supposing s (k-3) was memorized by s (k~2) and 71d of storage 
cellsrespectivelyin the next time k+1. Clear pulse CLR4 is sent from the 
horizontal scanning circuit 33a to 71 d of storage cells which have 
memorized the oldest signaland s (k~3) which is a front sampling signal 
is eliminated synchronizing with this. And synchronizing with the light 
sensing portion transfer pulse sent from the horizontal scanning circuit 
33a immediately after thatthe signal from the amplifier 52 is inputted 
and s (k+1) which is a sampling signal of new infrared light intensity 
is memorized by 71 d of storage cells. Whenever the time k of a sampling 
changes belowthe sampling signal of new infrared light intensity is 
rewritten one by one by the storage cell which has memorized the 
sampling signal of the oldest infrared light intensity similarlyand is 
memorized. 

[0049]The signal of a sampling of the infrared light intensity memorized 
by each storage cells 71a thru/or 71d is outputted in parallel with the 
matrix circuit 72. 

[0050]The matrix circuit 72 makes the switch 81a 84a and 81b thru/or 
84b81c or 84cand 81d thru/or 84d turn on and off based on the operation 
selection signal from the operation control part 35. Namelymake into the 
signals VI thru/or V4 here the signal outputted from each storage cells 
71a thru/or 71dand in a certain time k to the storage cell 71a. When s 
(k~3) shall be memorized by s (k) and the storage cell 
71brespectivelythe function f (k) which should be calculated is 
expressed with s (k-2) and 71d of storage cells to s (k-1) and the 
storage cell 71c by them by the following formulas (3). 
[0051] 

f(k) ={s(k)+s(k-l)}-{s(k-2) +s(k-3)}+BIAS .... (3) 
Namelyf(k) =V1+V2~V3-V4+BIAS .... (4) 
It becomes. 

[0052]And in the next sampling time k+lsince oldest sampling signal s 
(k-3) of the infrared light intensity of the time k-3 memorized by 71 d 
of storage cells is transposed to s (k+1) it is as follows. [ of the 
formula which should be calculated ] 
[0053] 

f (k+1) ={s(k+l) +s(k)}-{s(k-l) +s(k-2)}+BIAS .... (5) 
Namelyf(k+1) =V4+V1-V2-V3+BIAS .... (6) 
It becomes. 

[0054]And whenever the time of a sampling changesthe following four 
kinds of operations will be repeated. 



[0055] 

f(k) =V1+V2-V3-V4+BIAS .... (7) 
f (k+1) =V4+V1-V2-V3+BIAS . . . . (8) 
f(k+2) =V3+V4-V1-V2+BIAS .... (9) 
f(k+3) =V2+V3-V4-V1+BIAS . . . . (10) 

Thussince there is no necessity of transmitting the signal itself 
memorized by changing the combination of addition and subtraction one by 
one for every sampling timeand performing an operation between storage 
cellsdegradation of the signal produced in the case of transmission etc. 
can be controlled. 

[0056]Herethe compute modes of the above-mentioned formula (7) thru/or a 
formula (10) are defined as the modes A thru/or Drespectively. 
[0057]Hereit returns to explanation of drawing 6 . The matrix circuit 72 
controls the switch 81a 84a and 81b thru/or 84b81c or 84cand turning on 
and off of 81d thru/or 84d based on the compute mode of the operation 
selection signal transmitted from the operation control part 35. For 
examplewhen the operation selection signal in the mode A is transmitted 
to the matrix circuit 72 from the operation control part 35 at the time 
of sampling time kthe matrix circuit 72Carry out the switches 81a 
thru/or 84a to oneand the signal V2 memorized by the signal VI memorized 
by the storage cell 71a and the storage cell 71b is supplied to the plus 
input of the differential amplifying circuit 93 of the comparing element 
62The signal V4 memorized by the signal V3 memorized by the storage cell 
71c and 71 d of storage cells is supplied to the negative input of the 
differential amplifying circuit 93 of the comparing element 62. 
[0058]The load 91a of the comparing element 62 is connected to the plus 
input of the differential amplifying circuit 93the load 91b is connected 
to the negative input of the differential amplifying circuit 
93respectivelyand the current inputted from the storage cells 71a 
thru/or 71d is transformed into voltage. The variable current source 92 
generates bias currentsupplies it to the plus input of the differential 
amplifying circuit 93and is added to the signal inputted into the plus 
input of the differential amplifying circuit 93 among the storage cells 
71a thru/or 71d as BIAS in a formula (7) thru/or a formula (10). The 
differential amplifying circuit 93 calculates the difference of a plus 
input and a negative input. 

[0059]For exampleto the matrix circuit 72 of the storage parts store 61 
the operation selection signal in the mode A from the operation control 
part 35When inputtedthe switches 81a thru/or 84a are carried out to 
oneand the signal V2 memorized by the signal VI memorized by the storage 
cell 71a and the storage cell 71b is inputted into the plus input of the 



differential amplifying circuit 93 of the comparing element 62. The 
signal V4 memorized by the signal V3 memorized by the storage cell 71c 
and 71 d of storage cells is inputted into the negative input of the 
differential amplifying circuit 93 of the comparing element 62. 
Thereforethe differential amplifying circuit 93 will perform the 
operation of a formula (7). 

[0060] In the example of this explanationalthough the differential 
amplifying circuit 93 is used for the comparing element 62a chopper type 
comparison circuit can also be used. 

[0061]Nextoperation of the image processing device 1 is explained with 
reference to the flow chart of drawing 7 . 

[0062] In Step Slbased on the control signal from the system control part 
llpattern light (infrared light) is generated by the pattern light 
projection part 12and it glares towards the photographic subject 2. And 
it is condensed with the lens 13 and the infrared light and visible 
light which were reflected from the photographic subject 2 enter into 
the prism 14. 

[0063] In Step S2the spectrum of the light which entered is carried out 
to visible light and infrared light by the prism 14the visible light by 
which the spectrum was carried out enters into the imager 15and infrared 
light enters into the distance sensors 17respectively. 
[0064] In Step S3the imager 15 extracts the information on a color from 
visible lightand outputs it to the video signal processing section 16. A 
gain adjustment and color signal processing are performedand the 
information on a color that it was inputted into the video signal 
processing section 16 is outputted to the computer 19 as a color video 
image signal. On the other handthe distance sensors 17 receive infrared 
light by each of that pixel 41generate the binary-ized signal which can 
detect the peak of the intensityand output it to the formed data 
treating part 18. Processing of the pixel 41 of the distance sensors 17 
is mentioned later. The formed data treating part 18 asks for the 
sampling time which serves as a peak of infrared light from the binary- 
ized signal from the distance sensors 17by the principle of the infrared 
light intensity corresponding to the sampling time to 
triangulationcalculates the distance to the photographic subject 2and 
outputs it to the computer 19 as a formed data signal. 
[0065] In step S4after the computer's 19 compounding the color video 
image signal and formed data signal which were inputted and performing 
computer graphics processingoutput to the monitor 3or output to the 
external storage 4it is made to memorizeand processing is ended. 
[0066]Nextin the sampling time k-1 in the timing chart of drawing 9 with 



reference to the flow chart of drawing 8 and the timing chart of drawing 
9Vl=s (k-4)V2=s (k-l)V3=s (k-2)and V4=s (k-3) explain the operation in 
the case of memorizing to the storage cells 71a thru/or 71d of the pixel 
41 of the distance sensors 17 of drawing 2 as a sampling signal of 
infrared light intensity. 

[0067] In Step Sllif the reset pulse transmitted from the timing 
generator 32 immediately after the sampling time k~l is inputted into 
the light sensing portion 51the light sensing portion 51 will reset a 
light-receiving leveland will newly start light-receiving of infrared 
light (the inside of drawing 9 accumulation phase). 

[0068] In Step S12the light sensing portion 51 carries out photoelectric 
conversion of the infrared light which newly received light 
synchronizing with the light sensing portion transfer pulse (not shown) 
from the timing generator 32and outputs it to the amplifier 52. 
[0069] In Step S13synchronizing with the amplifier drive pulse (not 
shown) from the timing generator 32the amplifier 52 amplifies the signal 
inputted from the light sensing portion 51and outputs it to the 
operation part 53. 

[0070]The operation part 53 synchronizes with clear pulse CLR1 ( drawing 
9) from the horizontal scanning circuit 33a in Step S14The signal s of 
the storage cell 71a which is the oldest signal (k-4) is eliminatedand 
the storage cell 71a is made to memorize the signal from the amplifier 
52 synchronizing with storage parts store transfer pulse TX1 ( drawing 9 ) 
from the continuing horizontal scanning circuit 33a. 

[0071] In Step S15the storage cells 71a thru/or 71d output the memorized 
signals VI thru/or V4 to the matrix circuit 72. 

[0072] In Step S16the matrix circuit 72The one [ the switches 81a thru/or 
84a ] based on the signal in the mode A of the operation selection 
signal from the operation control part 35The signals VI and V2 are 
supplied to the plus input of the differential amplifying circuit 93 of 
the comparing element 62 among the signals VI thru/or V4 inputted from 
the storage cells 71a thru/or 71dand the signals V3 and V4 are supplied 
to a negative inputrespectively. 

[0073] In Step S17the signal which added BIAS supplied from the variable 
current source 92 to the signals VI and V2 inputted into the plus input 
from the matrix circuit 72and the signals V3 and V4 inputted into the 
negative input are changed into voltage from current with the loads 91a 
and 91b. The differential amplifying circuit 93 of the comparing element 
62 performs the operation of a formula (7) from these signals VI thru/or 
V4 and BIAS synchronizing with the comparing element drive pulse (not 
shown) from the horizontal scanning circuit 33a (the inside of drawing 



9operation phase). And the result of an operation is outputted to the 
outputting part 54. In the example of drawing 9 1 is outputted and it is 
shown that the peak of the infrared light intensity which received light 
was detected. 

[0074] In Step S18the outputting part 54 outputs the result of an 
operation as a pixel signal to the output circuit 34 via the common 
signal lines 42 synchronizing with the selection signal from the 
horizontal scanning circuit 33a (it corresponds to the sampling time k+1 
of a sampling synchronized signal) (the inside of drawing 9 output phase). 
[0075] In Step S19synchronizing with the control pulse from the timing 
generator 32the output circuit 34 outputs a pixel signal to the formed 
data treating part 18and ends processing. 

[0076]This processing is repeated for every sampling timeas shown in 
drawing 9 . Name lyal though the above-mentioned processing enters from the 
reset pulse immediately after the sampling time k~l at the accumulation 
phase of compute mode A and enters from clear pulse CLR4 at an operation 
phaseAt this timeit will enter at the accumulation phase of the 
operation in that following mode Band this is henceforth repeated by the 
reset pulse of sampling time k. 

[0077]Although the light sensing portion 51the amplifier 52the operation 
part 53and the outputting part 54 were formed in the pixel 41it may be 
made to put the operation part 53 and the outputting part 54 on the 
exterior of the pixel 41 in the above explanationas shown in drawing 10 . 
The example of drawing 10 shows the distance sensors 17 which carried 
out the operation part 53 and the outputting part 54 of each pixel 41 of 
the distance sensors 17 corresponding to drawing 2 every exception. 
[0078] In drawing lO the same numerals are given to the case of drawing 
2and the corresponding portionand the explanation is omitted suitably. 
In the distance sensors 17 of drawing lO the pixel output line 101 and 
the memory operation area 102 are newly formed. The pixel signal from 
each pixel 41 in the optical area 31 is outputted to the storage parts 
store 61 corresponding to each pixel 41 of the memory operation area 102 
via the pixel output line 101. Only the light sensing portion 51 and the 
amplifier 52 of drawing 4 are provided in each pixel 41 corresponding to 
drawing lO the portion which consists of the operation part 53 and the 
outputting part 54 after it is provided in the storage parts store 61 
corresponding to each pixel 41 of the memory operation area 102and both 
are connected to the pixel output line 101. Thusby carrying out every 
exceptionthe pixel 41 can be efficiently arranged to the optical area 31. 
[0079] Drawing 11 is a timing chart which shows processing of two or more 
pixels 41 of the optical area 31 of drawing lO and the memory operation 



area 102. Namelyfor exarapleby sampling time [ of drawing 9 ] kand k+1 
about the pixel i-1 the storage parts store 61 of the operation storage 
area 102The signal inputted into the storage cell (either 71a thru/or 
71d cell the data of old time is remembered to be) from which data was 
eliminated synchronizing with the clear pulse via the amplifier 52 and 
the pixel output line 101 synchronizing with the transfer pulse from the 
light sensing portion 51 is made to memorize. And the light sensing 
portion 51 corresponding to the storage parts store 61 is reset by the 
reset pulseand newly starts light-receiving. On the other handthe signal 
memorized by the storage cells 71a thru/or 71d performs an operation 
based on the operation selection signal inputted from the operation 
control part 35and outputs it to the outputting part 54. And the 
outputting part 54 is outputted to the output circuit 34 via the common 
signal lines 42 synchronizing with the selection signal from the 
horizontal scanning circuit 33a. Thenthe following pixel i performs 
processing shown in drawing 9 by the same sampling time k as the pixel 
i-land k+1. In drawing lO although the horizontal scanning type type case 
corresponding to drawing 2 was explainedthe same may be said of the 
vertical-scanning type type case of drawing 3 . 

[0080] In the above explanationalthough it was made to make an image 
processing device perform three-dimensional image processingthe image 
processing device can perform with a picture other processings which 
need data processing by changing arithmetic contents. For exampleit is 
good also as three-dimensional thermography by making it apply to the 
thermography etc. which measure temperature distribution with picture 
informationand combining above-mentioned three-dimensional image 
processing and thermography. 

[008l]As mentioned aboveimage processing in real time becomes possible 

by giving a calculation function to each pixel 41. 

[0082] 

[Effect of the Invention] Since the signal by which photoelectric 
conversion was carried out for every element of optical area was 
calculated under the predetermined rule according to an image processing 
device according to claim 1 and the image processing method according to 
claim 6data processing of the picture information in real time becomes 
possible. 
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[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the composition of the image 

processing device which applied this invention. 

[Drawing 2] It is a block diagram showing the composition of the 

horizontal scanning type distance sensors of drawing 1 . 

[Drawing 3] It is a block diagram showing the composition of the 

vertical-scanning type distance sensors of drawing 1 . 

[Drawing 4] It is a block diagram showing the detailed composition of the 
pixel of drawing 2 . 

[Drawing 5] It is a figure explaining the arithmetic method of the peak 
of the light received. 

[Drawing 6] It is a figure showing the detailed composition of the 
operation part of drawing 4 . 

[Drawing 7] It is a flow chart explaining operation of the image 
processing device of drawing 1 . 

[Drawing 8] It is a flow chart explaining operation of the pixel of 
drawing 2 . 

[Drawing 9] It is a timing chart explaining operation of the pixel of 
drawing 2 . 

[Drawing 10] It is a block diagram showing other composition of the 
distance sensors of drawing 4 . 

[Drawing 11] It is a timing chart explaining operation of the distance 
sensors of drawing 10 . 
[Description of Notations] 

1 An image processing device and 2 A photographic subject and 3 A 
monitor4 An external storagell system control partsand 12. A pattern 
light projection part and 13 A lens and 14 Prisml5 An imager and 16 A 
video signal processing sectionl7 Distance sensors and 18 formed-data 
treating partl9 A computer31 optical areaand 32. A timing generator and 
33a A horizontal scanning circuit33b A vertical scanning circuit and 34 
output circuits35 An operation control part and 4141aor 41n. A pixel42 
common signal linesand 51 A light sensing portion52 An amplifier and 53 
Operation part and 54 An outputting part61 A storage parts store and 62 
A comparing element and 71a thru/or 71d. A storage cell72 matrix 
circuits81a or 81d and 82a 82d and 83a thru/or 83d84aor 84d [ A 
differential amplifying circuit and 101 / A pixel output line and 102 / 
Memory operation area ] A switchand 91a and 91b Load and 92 A variable 
current source and 93 



